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an English Speaking Test 

Yuji Nakamura 



Theoretical background and rationale 

With the advent of the Communicative Approach, the role of speaking ability has 
become more central in language teaching. Consequently, performance testing, especially 
testing speaking ability, has become one of the most important issues in language testing. 

The increasing demand for the assessment of students’ English speaking skills in 
Japan has forced teachers to recognize certain limitations in our understanding; for 
example, the construct of speaking. Since we are not sure what exactly the components 
of speaking are, it is sometimes difficult to explain what test scores mean, especially in 
relative terms. 

Some scholars (Boldt and Oltman 1993; Henning and Cascallar 1992; Fulcher 1994) 
have approached the construct of oral proficiency, and explored the structure gradually. 
However, still there seems to be many limitations in this area because of the nature of 
speaking ability and the complexity of measuring spoken utterances. 

Purpose of the research 

With the goal of making a construct valid test of speaking ability in mind, the 
purpose of this research is to examine the following points: 

1) whether the proposed nine traits (pronunciation, grammar, discourse, fluency, 
content, vocabulary, comprehensibility, interactional competence, and sociolin- 
guistic competence) are relevant and separable parts of speaking ability. 

2) whether there is a relationship between methods and language ability. For this 
purpose, three methods (a writing test, an interview test, and a tape-mediated 
speaking test) were designed. 

3) whether the traits are fundamental elements of ability. 

4) how the traits (with the same name across the three methods e.g. grammar in 
writing, grammar in the interview, and grammar in the tape test) are related. 
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The research will also attempt to discover the extent to which the proposed con- 
struct of speaking is reflected in other standardized tests such as the Test of Spoken 
English(TSE) or the American Council on the Teaching of Foreign Languages Oral Pro- 
ficiency Interview (ACTFL OPI). 

Research design and methods 

Twenty-nine college students took the set of tasks (a writing test, an interview test, 
and a tape- mediated speaking test). Seven English teachers, who are all native speakers 
of English, scored the test results using the 1-4 point scale rating sheet. Among the 
seven teachers, two of them rated the writing results, another two rated the interview 
test results, and the other three rated the tape-mediated test results. 

Establishment of Research Instrument Procedure 

1) The operational definition of each trait is given below. 

The general outline of the operational definition is whether a speaker can express 
effectively him/herself by using the following eight sub-traits 

(1) pronunciation 

segmental features (individual sounds; vowels and consonants) 

supra-segmental features (stress, rhythm, intonation) 

volume (audible or not) 

(2) grammar 

tense and aspect 

noun- verb agreement 

word order 

noun-personal pronoun agreement 

(3) vocabulary 

variety of words 

word choice 

idioms 

nuances 

(4) content 
creativity 

imaginativeness 
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(5)-(a) fluency for speaking 

ease of speaking 

speed 

length, place and frequency of pauses 

(5) -(b) fluency for writing — intelligibility 
how easy the composition is to read 

how smoothly raters can read through the sentences 

(6) discourse 
cohesi ven ess 

logical combination of sentences 

(7) interactional competence 

ability to use conversationally formulaic expressions 

(8) sociolinguistic competence 

ability to express his or her ideas in a sociolinguistically appropriate manner 

(9) comprehensibility 

the degree of intelligibility to the raters or overall impression 
* — means examples of each trait 

2) Making a matrix of traits and methods 
*• • - means cannot be measured 

** method in this research means task (writing, an interview, a tape-mediated 
speaking test) 



Method 







Writing/Inter view/Tape-mediated 
test test speaking test 


1) 


Trait 

pronunciation 




2) 


grammar 




3) 


fluency 




4) 


discourse 




5) 


content 




6) 


vocabulary 




7) 


interactional 




8) 


competence 

sociolinguistic 




9) 


competence 

comprehensibility 
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Making each test 

1) Writing Test 

Students will be given a cartoon or a comic strip and given 15 minutes to write a 

composition of 100 words about the cartoon. There are two types of cartoons. One 
is a narrative type for evaluating students’ grammar, fluency, discourse, content, 
vocabulary. The other is a conversation type for evaluating students’ interactional 
competence and sociolinguistic competence. 

2) Interview Test 

A 5- minute conversation with the clcissroom teacher following the suggested pro- 
cedure below. (N.B. This test is video-taped, so a VCR machine is needed.) The 
teacher will choose some of these questions which are relevant to probe each stu- 
dent’s speaking ability. 

1) Greeting 

Self-introduction( Plecise introduce yourself by giving your name, the name of 
your high school and your hobbies.) 

2) Short answer interview. 

Where do you come from? 

Do you have any club activities? 

Do you have any brothers or sisters? 

How do you come to school? 

How long does it take to come to school? 

Do you do a part-time job? 

3) Long answer interview 

Can you tell me more about your high school? 

How many students are there? 

Where is it located? 

Can you tell me more about your family? 

What do your family members do? 

Can you tell me more about your club activities? 

How many members are there? 

How often do you meet with them? 

Can you tell me more about your hobbies? 

Why do you like them? 
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When did you start? 

Tell me more about your part-time job. 

What is your job? 

What do you actually do? 

What is your favorite TV program? 

What kind of program is it? 

Why do you like it? 

What is the most interesting news these days? 

Why do you think it is interesting? 

What do you think about it? 

4) Free form 

Tell me more about your life at college 

Do you talk to teachers?-What are the teachers like? 

Do you ever eat lunch at the dining hall?-What is 
the dining room like? 

Do you go to the libraxy?-What is the library like? 

5) Winding up 

Note: 

We needed something to elicit students’ sociolinguistic competence and interac- 
tional competence, such as a role play. However, it would interrupt the natural 
flow of conversation between the interviewer and the student. Interviewers should 
include natural/authentic questions in order to measure students’ interactional 
and sociolinguistic competence in the greeting section and the winding up section 
so that interviewers do not need to stop the natural flow of conversation to start a 
role play abruptly. It is possible to say that the teacher-student conversation is one 
type of sociolinguistic context. Students’ sociolinguistic ability can be measured 
through their ability to use formal expressions or polite expressions in the inter- 
view context. It may also be true that interactional ability can be rated through 
students’ ability to handle necessary formulaic expressions in their responses to 
their interviewers. 
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Conclusion for the note. 

Any test is a sample of the whole language proficiency. The interview test is a 
sample of the whole language proficiency although it restricts the great possibilities of the 
measurement. Interview tests can be considered as samples of the whole sociolinguistic 
and interactional context in the real world. Therefore, the interview test is valid as a 
test tool to measure students’ language proficiency. 

We can think of the questions and responses that appear in the interview session as 
samples of sociolinguistic and interactional things in the restricted context of a teacher- 
student relationship. 

They are having an interaction in the interview. So their responses can be measured 
in terms of interactional ability. If students can interact with interviewers using formulaic 
expressions and polite expression appropriately, then we think students can also have 
sociolinguistic/inter actional competencies. 

3) Tape-mediated speaking test (See Appendix) 

speech making test 

interactional competence test 

sociolinguistic competence test 

Administration of the Test 

1) Subjects 

29 students of Tokyo Keizai University 

2) Test administration 

Tests were conducted in class. 
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Rating procedure and rating scale 

1) A writing test wets conducted by the researcher and evaluated by two raters(native 
speakers of English). 



Rating Scale for Writing 

( means cannot be measured) 



12 3 4 

poor good 



1) grammar 

2) discourse 

3) content 

4) vocabulary 

5) *fluency 

6) interactional 
competence 

7 ) sociolinguistic 
competence 

8) pronunciation 

9) overall impression (comprehensibility) 

* fluency in writing means how easy it is to read, and how 
smoothly raters can read through it. 



2) Interview tests were administered by two cleissroom teachers (native speakers of 
English) and evaluated by themselves. 



Rating Scale for Interview 



12 3 4 

poor good 



1) pronunciation 

2) grammar 

3) discourse 

4) content 

5) vocabulary 

6) fluency 

7) interactional competence 

8) sociolinguistic competence 

9) overall impression (comprehensibility) 
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3) Tape-mediated tests were administered by the researcher and evaluated by three 
raters (native speakers of English), 



Rating Scale for Speaking Test 





1 2 3 

poor 


4 

good 


1) 


pronunciation 




2) 


grammar 




3) 


discourse 




4) 


content 




5) 


vocabulary 




6) 


fluency 




7) 


overall impression (comprehensibility) 




8) 


interactional competence test 




9) 


sociolinguistic competence test 





Data Analysis 

Factor analysis w£is adopted 1) to examine the categorization of traits, 2) to inves- 
tigate the relationship between the method and the factor, 3) to explore the degree of 
contribution of the trait in each factor. 

Inter-trait correlation coefficients within each method were calculated in order to 
check the reciprocity of each trait internally within each factor. Also the correlations 
over three methods were examined to show the relationship among the traits. 

The correlations among three methods were computed to investigate the relationship 
among them. 

Results ajid Discussion 

1. Results of Factor Analysis (Principal Component Factor Analysis) 

Table 1 shows that three factors were extracted through the factor analysis, and these 
three factors agree with the different methods designed previously for this research as 
follows: 
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Table 1 Results of Factor Analysis 



trait 


Factor 1 


Factor 2 


Factor 3 


Commu- 

naiity 


icompre 


0.93 


0.12 


0.11 


0.89 


idiscour 


0.9 


0.13 


0.06 


0.83 


ifluency 


0.82 


0.24 


0.21 


0.78 


iso-com 


0.81 


0.08 


0.16 


0.69 


igramm 


0.75 


0.13 


0.22 


0.64 


ivocabul 


0.65 


0.38 


0.19 


0.61 


iin-com 


0.65 


0.29 


0.1 


0.52 


ipronun 


0.62 


0.36 


0 


0.51 


icontent 


0.61 


0.3 


0.1 


0.47 


tdiscour 


0.17 


0.93 


0.03 


0.89 


tcontent 


0.22 


0.84 


0.12 


0.76 


tcompre 


0.31 


0.84 


-0.01 


0.8 


tvocabul 


0.26 


0.83 


0.09 


0.76 


tgramm 


-0.06 


0.82 


0.22 


0.73 


tfluency 


0.33 


0.79 


0.1 


0.74 


tpronun 


0.36 


0.71 


0.22 


0.68 


tso-com 


0.29 


0.55 


0.32 


0.49 


tin-corn 


0.28 


0.47 


0.43 


0.49 


wcompr 


0.2 


-0.03 


0.89 


0.84 


wdiscou 


0.07 


0.24 


0.84 


0.76 


wconten 


0.23 


0.23 


0.81 


0.77 


wgramm 


-0.02 


0.05 


0.79 


0.64 


win-corn 


0.05 


0.28 


0.79 . 


0.7 


wfluenc 


0.04 


0.25 


0.78 


0.68 


wvocabu 


0.41 


0.09 


0.66 


0.6 


wso-com 


0.17 


-0.12 


0.61 


0.41 


eigen val 


10.83 


3.85 


3 




pet. of var 


41.7 


14.8 


11.5 




cum. pet 


41.7 


56.5 


68 





Note: 

In the column of trait, “i” means an interview, “t” means a tape test and “w” 
means writing. 

1) Factor 1 (Direct Oral Communication Ability) (DOCA) 

Items for this factor are comprehensibility, discourse, fluency, sociolinguistic com- 
petence, grammar, vocabulary, interactional competence, pronunciation and con- 
tent. We will name this feictor Direct Oral Communication Ability. This ability 
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deals with students’ speaking ability in the face to face context, or students’ re- 
ciprocal ability in the live context. The students utilize these linguistic elements 
for communication with some strategies such as eye contact or facial expressions. 
Therefore, students work on non-verbal signs as well as verbal messages. 

2) Factor 2 (Semi-Direct Oral Communication Ability) (SDOCA) 

Items for this factor are discourse, content, comprehensibility, vocabulary, gram- 
mar, fluency, pronunciation, sociolinguistic competence and interactional compe- 
tence. Factor 2 is named Semi-Direct Oral Communication Ability. This ability 
concerns students’ speaking ability without being influenced by the external or 
the non-verbal elements. Students can/should focus on the monologue type com- 
munication by expressing their own ideas, or they can perform in English by con- 
centrating on the in-coming information through their ears. 

3) Factor 3 (Written Communication Ability) (WCA) 

Items for this factor are comprehensibility, discourse, content, grammar, inter- 
actional competence, fluency, vocabulary and sociolinguistic competence. We can 
call Factor 3 Written Communication Ability because students could communicate 
with each other through the written form. Students should take into consideration 
the context or the situation in which they are supposed to express their ideas. 

The reason I have used the term “communication” for these three abilities is 
that students could communicate with each other in any of the three ways, by 
using verbal messages, by adopting non-verbal messages or by taking the written 
form. All are related to communication. 

In the table, if there is a communication ability per se, the cumulative percent- 
age of the variance suggests that 68% of the whole language ability ( “communica- 
tion ability”) can be explained by these three factors. The remaining 32% which 
cannot be clarified by the present results may involve reading ability, listening 
ability as well as strategic skills or cultural awareness which could be composite 
of the whole language ability. 
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Correlation Coefficients of Three Methods (Abilities) 



Table 2 Correlation of Three Abilities 





writing 


interview 


tape- test 


writing 

interview 


0.36 

(p=.052) 






tape- test 


0.38 

(p=.042) 


0.55 

(p=.002) 





(n=29) 



1) There is very low correlation between writing (WCA) and an interview (DOCA), 
and between writing and a tape test (SDOCA), In other words, writing is quite 
different from interview and tape testing in the sense of a test mode. An interview 
and a tape test require audio dealing ability while writing does not. Therefore, 
it is not surprising that there is a comparatively high correlation between tape 
test and interview, because they both cope with students’ oral proficiency. What 
we could conclude from this result is that although three factors (WCA, DOCA 
and SDOCA) were obtained through the factor analysis, writing ability (WCA) is 
a different one from DOCA and SDOCA, which is quite understandable and not 
surprising, 

2) An interview (DOCA) and a tape test (SDOCA) are distinct because some students 
feel nervous about speaking to a machine just mechanically, while other students 
do not feel easy when they must talk to an interlocutor in the interview session 
even with their teachers. Thus, the test practice can affect the difference of the 
two abilities. 

Investigation of Individual Traits 

Since there were no irregular traits in the factor analysis for constructing the factors, 
each trait in each factor was contributing well to form the reasonable factor. Also, they 
(the nine traits in each factor) were relevant as components of each factor. The question 
whether the traits were separable or not could be clarified through the following inter- 
trait correlations in Table 3 (Table 3-1 to Table 3-3). 
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Table 3 Inter*Trait Correlations 
Table 3-1 Inter-Trait Correlations in Writing 





gram 


disco 


cont 


voca 


flue 


inter socio- comp 


grammar 

discourse 


0.74 












content 


0.58 


0.82 










vocabular 


0.48 


0.6 


0.64 








fluency 


0.64 


0.73 


0.74 


0.64 






inter- comp 


0.61 


0.67 


0.73 


0.54 


0.65 




socio-comp 


0.51 


0.35 


0.35 


0.4 


0.36 


0.58 


comprehensibility 


0.67 


0.81 


0.81 


0.68 


0.68 


0.7 0.54 



Table 3-2 Inter-Trait Correlations in Interview 





pron 


gram 


disco 


cont 


voca 


flue 


incom socm comp 


pronunciation 
















grammar 


0.59 














discourse 


0.56 


0.68 












content 


0.55 


0.52 


0.61 










vocabular 


0.51 


0.5 


0.7 


0.52 








fluency 


0.47 


0.71 


0.83 


0.59 


0.65 






inter-comp 


0.46 


0.52 


0.72 


0.65 


0.53 


0.76 




socio-comp 


0.66 


0.75 


0.69 


0.47 


0.57 


0.7 


0.44 


comprehensibility 


0.62 


0.7 


0.83 


0.57 


0.73 


0.79 


0.67 0.79 



Table 3-3 Inter- Trait Correlations in Tape Test 





pron 


gram 


voca 


conte 


flue 


disco 


compr 


int-c 


socio- 


pronunciation 




















grammar 


0.62 


















vocabular 


0.68 


0.65 
















content 


0.69 


0.7 


0.8 














fluency 


0.63 


0.55 


0.74 


0.76 












discourse 


0.68 


0.66 


0.78 


0.71 


0.83 










comprehensibility 


0.72 


0.59 


0.78 


0.81 


0.88 


0.77 








int-comp 


0.63 


0.54 


0.42 


0.52 


0.41 


0.38 


0.36 






socio-comp 


0.53 


0.56 


0.65 


0.56 


0.51 


0.47 


0.52 


0.68 
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In this table, as the inter-trait correlations in writing(Table 3-1: the range is from .35 
to .82), those in an interview test ( Table 3-2:the range is from .44 to. 83) and those 
in a tape test (Table 3-3:the range is from .36 to .88) show, the traits are all mutually 
interrelated and mutually exclusive. Therefore, it might be possible to say that they are 
separable. 

This is beyond the scope of the factor analysis; however, a little investigation of the 
same-trait different- method analysis demonstrates that correlations of the same traits 
across the three methods are not as high as expected. (See Table 4) 



Table 4 Correlations of the same traits across the three methods 
Table 4-1 Correlations of the same traits between Writing and Interview 



wgram wdis wcon wvoc wflue win- wso- wcomp 

com comp 



igramm 0.2 

idis 

icont 

ivoc 

iflue 

iin-com 

iso-com 

icompr 



0.13 



0.18 



0.49 



0.2 



0.26 

0.21 



0.32 



Table 4-2 Correlations of the same traits between Writing and Tape-Test 



wgram wdis wcon wvoc wflue win- wso- wcomp 

com comp 



tgramm 

tdis 

tcont 

tvoc 

tflue 

tin-corn 

tso-comp 

tcompr 



0.11 



0.12 



0.33 



0.17 



0.31 



0.47 



0.31 



0.05 
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Table 4-3 Correlations of the same traits between Interview and Tape- Test 





igram 


idis 


icont 


ivoc 


iflue 


iin- 


iso- 


icompr 


ipron 














comp 


comp 






tgramm 

tdis 


0.11 


0.21 
















tcont 

tvoc 

tflue 

tin-comp 

tso-comp 

tcompr 

tpronu 






0.29 


0,48 


0.44 


0.27 


0,29 


0.39 


0.62 



N.B.: gramm=grammar; dis=discourse; cont=content; 

voc=vocabulary; flue=fluency; in-comp=interactional 
competence; so-comp=sociolinguistic competence; 
compr=comprehensibility; pronun=pronunciation 



One possible reason for the low inter- trait correlations is that traits in each method 
are internally influenced by the nature of the factor and eventually presented the low 
correlation against the other traits in the other methods even though they have the 
same naming. Another possible reason is that the original definition of each trait might 
not be clear. Still another possible reason is that the trait in each method was basically 
different. In other words, there is a great possibility that grammatical ability in spoken 
form is different from grammatical ability in written form. 

Comparison among an Interview Test, a Tape Test, ACTFL and TSE 
Let us take a look at the components of the four tests before we compare them. 
Components of an Interview Test 

pronunciation, grammar, vocabulary, fluency, discourse, content, comprehensibility, 
interactional-competence, sociolinguistic-competence 
Components of a Tape Test 

pronunciation, grammar, vocabulary, fluency, discourse, content, comprehensibility, 
interactional-competence, sociolinguistic-competence 
Components of ACTFL 

pronunciation, grammar, vocabulary, fluency, pragmatic competence(=confident use 
of various conversation management devices), sociolinguistic competence (appropriate 
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use of the major registers) 

N.B. The idea of pragmatic competence is similar to that of interactional compe- 
tence in the present research, and the idea of sociolinguistic competence resembles 
that of sociolinguistic competence in the present research. 

Components of TSE 
former version 

pronunciation, grammar, fluency, overall comprehensibility 
revised version 

communicative language ability (communicative effectiveness) 

comprehensibility 

linguistic accuracy (pronunciation, grammar, fluency, vocabulary) 

coherence and use of cohesive devices 

appropriateness of response to audience/situation 

addressing the communicative task 

Judging from the results of this research, an interview test and a tape test are measuring 
the whole communication ability from different modes. They might cover some common 
linguistic areas while there are some distinct areas that the counter part cannot com- 
pensate for. This might be true for the situation between ACTFL and TSE. ACTFL 
through an interview test and TSE through a tape test seem to play a similar role as 
the interview test and the tape test do in the present research for measuring the whole 
communication ability. 

Since, through this research, we were able to find the indispensable role played by 
both the direct and the semi-direct speaking tests, it is not as difficult to presume that 
ACTFL and TSE will function in the same way because of the similax components of 
these speaking tests. 

Conclusions 

Three factors obtained from the factor analysis corresponds with three methods 
designed for measuring the ability. These three factors were named Direct Oral Commu- 
nication Ability, Semi-Direct Oral Communication Ability and Written Communication 
Ability individually according to the nature of each ability. 

As far as speaking ability is concerned, the two modes mentioned above are rather 
distinct judging from the factor analysis. However, both are measuring the common area 
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of communication ability with the level of .55 correlation coefficients. 

Nine traits in Direct Oral Communication Ability (Interview Test) contribute to 
become the fundamental elements of the factor while they still maintain their own char- 
acteristics which cannot be replaced by other traits. This idiosyncrasy was made clear 
through the inter-trait correlation coefficients. The same is true for the nine traits in 
Semi-Direct Oral Communication Ability, The nine traits are functioning as factor con- 
struct elements but they still have their distinctiveness. 

The traits in each factor, even though they have the same naming (grammar, vo- 
cabulary, discourse etc.), are working separately from the other two in each factor. One 
reason for this phenomenon is that the trait, influenced by the other traits internally 
within the factor, produced a new phase of the trait. Another reason is that the trait in 
one factor is basically different from the other two similar named traits in other factors, 
even though they have the same name. In other words, it would be possible to coin the 
terms such as “grammar in face to face situations,” “grammar in the tape test situa- 
tion.” The solution to this problem is left in the results of a well-organized multi-trait 
multi-method analysis. 

One way to look at the low correlation among the methods can be explained as 
follows: The lower or intermediate-lower level students do not necessarily demonstrate 
their competence in all language abilities. Some are poor at writing, some are good at 
interview tests and others are skillful in speaking without an interlocutor. It is said that 
unlike these lower or intermediate lower students, advanced or high intermediate level 
students show their constantly high language competence in various language skills. This 
fact might be one of the causes of the lower inter- method correlations. 

Interview tests could measure part of the whole communication ability, and tape 
tests could measure part of the whole communication ability. The traits in each method 
can be a good composite of the ability the method is measuring. 

We were able to find some similarities between the present interview test and ACTFL 
interview test, and between the present tape mediated test and TSE. Further research 
with the concurrent validity will enforce the relationship among these tests. 

Finally, the following points should be taken into account for future study. 

1) The number of testees should be increased to get more generalizable results. 

2) A much higher inter-rater reliability within each method should be obtained. 

3) More exhaustive study of traits and methods through multi-trait multi-method 
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analysis should be undertaken. 



Note: 

This research was conducted by the financial support of Tokyo Keizai University 
Research Grant (PC 11-95). 
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Establishing the Construct Validity of an English Speaking Test 



Appendix 

1) Speech Making Test 

Please choose one topic you want to talk about among the eight topics given below 
and give a two-minute speech about it. (my friends, my family, part-time work, my 
hobbies, traveling fashion telephone conversations, college life) 

2) Interactional Competence Test 

Please give a quick and appropriate response in English to each sentence or question 
which will be heard through your headphones. 

Nice to meet you. 

What is your name? 

Could you spell it please? 

How are you? 

What do you do? 

Can you tell me the time? 

What is the date today? 

What is the weather like today? 

What do you usually do on Sundays? 

How do you come to school? 

3) Sociolinguistic Competence Test 

Please give an appropriate response in English in each context. 

Context 1 (Apologizing and making an excuse) 

Context 2 (Complaining and requesting) 

Context 3 (Asking for repetition) 

Context 4 (Interrupting) 

Context 5 (Telephoning: taking a message) 
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